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ABSTRACT 

This report summarizes the results of a study to 
determine the overall effectiveness of adjunct programming techniques 
and to determine further whether the form and frequency of self-test 
questions contained in the adjunct program have differential effects 
on learning. Four formats of adjunct programs were used by different 
groups as a supplement while reading text on military justice. A 
fifth group read the text without the aid of an adjunct program. 
Results are discussed in terms of short and long term effects on the 
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SUMMARY 



Piroblem 

Much of the current emphasis on improvement of instruction involves the careful analysis of training 
objectives and the restructuring of instructional materials to improve learning. Although this approach is 
effective, it requires considerable manpower and additional resources which in some cases are beyond the 
immediate c^abilities of local training organizations. An alternate approach, where applicable, is to 
augment existing training materials with adjunct questions designed to prom<^ more focused study and 
provide improved informational feedback to the student. This study was conducted to determine the 
effectiveness of this adjunct programming approach in learning a regularly taught ROTC subject, and to 
determine whether varying the format of the adjunct questions produced differential effects on learning 
and retention. More specifically, the purpose of this study was to: (1) investigate the effectiveness of 
adjunct questions, presented within the context of adjunct programming, in enhancing general retention of 
written materials, and (2) explore the differential effects of adjunct question mode (multiple choice versus 
constructed response) and ac^unct question pacing (by the page versus by the chaptex) oa short and long 
term retention when reading time is held constant. 

Approach 

The approach involved three m^or steps: (1) literature review, (2) development of adjunct program 
booklets, and (3) experimental evaluation. 

Literature review. The literature review indicated that there have been two main categories of rdevant 
research: (1) that dealing with the overall effectiveness of adjunct programming which consisted of 
adjunct questions with provisions for immediate feedback, and (2) that dealing with how adjunct 
questions influence reading behavior which in turn contributes to learning. 

The review revealed that previous demonstrations of the usefulness of adjunct progranmiing involved 
the use of mechanical or other specially constructed devices to provide immediate knowledge-of-results. 
The approach in the present study was to provide knowledgeof-results by simply printing the answer on 
the page immediately fdlowing the adjunct question. 

The previous research, dealing with how adjunct questions "influence reading behavior, has been 
typified by variations of adjunct question position, mode, pacing, etc., under carefully controlled 
laboratory conditions using short term retention tests. Students were allowed to read the materials ordy 
once, renting in different lengths of time spent by experimental and control groups in studying the 
materials. The approach taken in this study was to control "time-to-study" in a realistic classroom 
environment, using both immediate and delayed retention measures. 

As a result of the review of literature, the answers to the following questions were sought under the 
conditions of a realistic classroom environment, equal *Hime-to-$tudy" and simple instructional feedback. 

(1) Do adjunct questions enhance general retention of written materials? 

(2) Do adjunct questions enhance relevant learning? 

(3) Do adjunct questions enhance incidental learning? 

(4) Does adjunct question mode or pacing affect retention? 

(5) Does adjunct question mode affect performance as a function of the question mode of the 
criterion test? 

(6) Are there differences in the effects of the above variables as a function of whether the test is 
immediate or delayed? 

Development of Adjunct Frogram Booklets. Four (4) different adjunct program booklets were 
developed consisting of a comprehensive series of questions highli^ting the major teaming objectives in the 
text. Knowledge of results in the form of correct answers was provided on the page following each 
question. The 59 questions in each booklet covered the main points of an existing text used in a regulariy 
taught ROTC subject.. Two of the booklets contained multiple choice adjunct questions, while the other 
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two were made up of equivalent constructed response questions. In addition, two of the booklets presented 
questions after each page of text whfle the other two presented questions at the end of each chapter. 

Experimental Evaluation 

Four experimental groups (with 29 subjects each) used one of the four adjunct program booklets 
while reading the basic text. A control group (n = 29) read the text witliout a booklet. All subjects were 
allowed 200 minutes to study the materials during normally scheduled classes. The 80-item criterion test 
consisted of five subtests. Four of the subtests varied in question mode (multiple choice versus constructed 
response) and question relevarKe (incidental versus relevant). Incidental questions were those which had not 
previously appeared as adjunct <iuestions. Relevant questions were those which had been used as adjunct 
questions. The fifth subtest consisted of general items, designed to measure the ability of subjects to apply 
information learned from reading to practical situations, llie total test was administered twice; one day 
after the last reading session and again 9 to 10 days later. 

Data from the experimental evaluation were analysed separately for the short and long-term retention 
measures using to al test scores, as well as various sub test scores. 

Results and Conclusions 

1. The adjunct programs in this study were an effective supplement to existing texts. The beneficial 
effects (as demonstrated by hi^er total test scores for those groups using the adjunct program booklets) 
held up under conditions of equal study time for control groups and the use of a simple printed 
knowledge-of-results approach, as opposed to a mechanical presentation device to provide instructional 
feedback. 

2. The beneficial effects of adjunct questions were limited, in large part, to the learning of relevant 
materials or those materials to which a specific adjunct question had been addressed during learning. 

3. There was no evidence that the adjunct questions enhanced incidental learning, under the 
conditions of this study. 

4. There was an interaction effect between adjunct question mode and test question mode. Where 
the test performance required recall rather than recognition of relevant materials, constructed response 
adjunct questions appeared to be superior to multiple choice questions. 

5. For the type of reading materials used ip the present study , placement of adjunct questions at the 
end of the chapter was as effective as interspersing questions after each page. 

6. The effects of adjunct questions remained relatively stable when measured by short and long term 
retention tests. 

Implications 

1. Adjunct programs are effective in focusing attention on key points while reading, and should be 
considered for use to augment existing course materials. 

2. Since adjunct programming does not require restructuring or re-designing basic learning materials, 
it is particularly suitable for enhancement of on^going courses where texts are already in existence. 

3. The resources requirements and programming techniques, necessary to produce adjunct programs, 
are relatively simple and well within the capabilities of local training organizations. 

4. Adjunct programs should include a comprehensive series of self-test questions to cover all 
materials which are to be retained. The process of identifying all major points in a text will also assist the 
instructor in clarifying to himself the learning objectives, which he desires the student to attain. 

5. Adjunct programs appear particularly uitable for use by the student in self-study at home. This 
application should be investigated further. 
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THE EFFECTS OF PACING AND MODE OF ADJUNCT QUESTIONS ON SHORT 
AND LONG TEm RETENTION OF WRITTEN MATERIALS 



U INTRODUCnON 

In the late 1920*s. Pressey (1926. 1927) first developed a simple mechanical device which 
administered and scored multiple choice questions automatically and also provided the student with 
immediate knowledge of results, tie felt that this approach had the potential for not only relieving the 
teacher of the mundane task of scoring tests, but also for use as a self-instructional device which 
incorporated the basic "laws*' of learning. His basic technique, consisting of the use of self-test or adjunct 
questions with immediate knowledge of results, is now commonly referrred to as adjunct programming or 
adjunct auto instruction (Briggs, 1968). 

Although Pressey's concepts were demonstrated in a series of studies (Pressey, 1950), it was not until 
the now classic articles of Skinner (1954, 1958) that the **industrial revolution in education*' envisioned by 
Pressey (1932) occuned. Skinner*s articles precipitated a phenomenal surge in instnictional technology in 
general, and in particular, the application of programmed instruction (PI) techniques in education and 
training. Briggs and Angell (1964) reported that there were almost 400 PI texts avaflable in various subject 
areas in 1963. In 1970, an Air Force report (HAF-T34, 1970) listed over 700 PI texts as being available in 
the Air Force training environment alone. 

In spite of the obvious impact of Skinner*s concepts, they have not gone unchallenged, both from 
theoretical and practical viewpoints. Skinner's basic notions of very small steps, active response and 
immediate kuowledge ot results have generated much experimentation. Briggs and Angell (1964), Anderson 
(1967), and Gagne and Rohwer (1969) have summarized much of the research results. Pressey (1963, 1967) 
as wen as others object to the fractionization of learning into small steps and advocate more meaningful 
groups of larger ^'frames*' or segments of instruction. From a practical point of view, Pressey (1967) notes 
the expense and lack of efficiency of programmed instruction in certain applications and views adjunct 
autoinstruction as a worthwhile alternate. 

A recent line of investigations precipitated bf Rothkopf (1965) has added further support to 
Pressey*s views on the effectiveness of using self-test or adjunct questions to enhance learning .'Originating 
from his interest in the role of student responses ih programmed instruction, Rothkopf (1963) became 
interested in how adjunct questions affect the inspection activities of the student while reading. On the 
basis of his research, Rothkopf has proposed that adjunct questions modify the s'udent*s mathemagenic 
activities, or those overt and covert activities of the student in ♦he instructional setting that influence the 
amount learned. Mathemagenic activities may include observable behaviors beneficial to learning, such as 
asking questions and looking at the instructor, or it may include behaviors determinental to learning such as 
yawning, daydreaming, etc. It may also involve covert activities tltat affect attentional or other mediating 
processes. These must be infened from their effect on observable behavior. Adjunct questions or "test-like'* 
events are considered by Rothkopf to be an environmental control which shapes a subject's mathemagenic 
behavior virile reading, which manifests itself in modified reading behavior (Rothkopf, 1965, 1970, 1972). 

Very much in consonance with Pressey 's (1963) views, Rothkopf accepts instructional materials as 
given, within limits, and emphasis is placed on promoting those activities of the student which allow him to 
achieve specified objectives using available instructional materials. As Rothkopf points out:. "The concept 
of mathemagenic activities tends to shift emphasis from investments of resources in the development of 
instructional materials to investment in the instructional environment. This is to a certain degree contrary 
to the current Zeitgeist which has been characterized by very careful design of instructional materials.*' 
(Rothkopf, 1970, p. 334.) 

In addition to being of considerable theoretical interest in the understanding of the mediating 
processes involved in instruction, the potential of adjunct questions to enhance training has substantial 
practical implications. If this technique is adaptable to specified training situations using existing 
instructional materials, it provides a very attractive alternative to other techniques where emphasis is on 
restructuring of learning materials and the use of special presentation media such as programmed texts and 
audio visual devices. 



Potential advantages of adjunct autoinstruction over programmed instruction may be summarized as 
foDows: 

1 . It makes use of existing instructional materiak and does not require major restructuring of subject 
matter. 

2. It does not require that the author have extensive knowledge of instructional programming 
technk}ues. 

3. It is less time consuming, both for the author * onstructing the material and the student in 
completing the program. 

Btdcground 

Description and Definition, Adjunct autoinstruction or adjunct progninuning is basically a series of 
carefully sequenced "self-test" questions which are designed to elk:it responses meeting stated instructional 
objectives. Although various formats have been used, th'* basic approach involves questions based on an 
existing text or training material, provisions for feedback, and provisions for remedial information if the 
initial respons: is incorrect. The approach normally requires that the learner: (I) read a portion of the 
study material, (2) answer a sdftest item or series of items covering the main points of the material, (3) 
obtain knowledge of results regarding the correctness of the answer, (4) refer to the material (or receive the 
correct answer) when the initial response is iricorrect, and (5) proceed to the next segment of instruction 
when he is able to answer all of the questions. Hgure 1 depicts the adjunct autoinstruction modd. 

Siegel and Fischl (1966) in defining adjunct autoinstruction contrast it to conventional programmed 
instruction: "Adjunct autoir^struction is, as its name implies, self instmction as an adjunct to some primary 
teaching method. It is a form of programmed instruction in that it: (1) i$ self pacing, (2) keeps the learner 
active in the learning process, and (3) provides the learner with inunediate feedback concerning his 
knowledge of the subject-matter. It differs from other forms of programmed instruction in that it is not 
intended to carry the full burden of teaching the lessonrrather, it follows the initial presentation of the 
lesson and treats only those parts of the lesson requiring clarification or empha^." 

Relevant Literature, Relevant literature pertaining to the use of adjunct questions may be generally 
categorized under two types: (1) that dealing with the overall effectiveness of adjunct autoimtnicUon as 
opposed to other methods, and (2) that in which the major emphasis has been the experimental 
inanipulations of adjunct questions to determine their influence on retention. A third category of relevant 
literature described in this review is drawn from the area of programmed instruction. 

Effectiveness of Adjunct Autoinstruction, TTie substantial number of studies, conducted by Pressey 
and his students, has provided data on the effectiveness of adjunct autoinstruction in teaching various 
college courses, using various mechanical devices and methods of providing knowledge of results. Much of 
this research conducted in the late 1940*s is summarized by Pressey (1950)- 

Two earlier studies involved initial efforts in the use of mechanical devices in the classroom 
environment, little (1934) conducted one of the first studies to demonstrate the effectiveness of using 
automatic scoring and self-testuig devices in the classroom environment. The first two experimental groups 
used two separate types of machines. The '*testing" machine automatically scored and provided the student 
with total score information. The "drill** machine allowed immediate knowledge of results on each 
question, and provided an opportunity for drill until aD questions were correctly answered (retained 
method). A third group (control), used simple answer sheets which were graded by the instmctor. For every 
two of 14 teaching units, 30 adjunct questions were provided. Analyses of the data revealed that the "driU** 
group performed better than the "test" group on both a 200 item multiple choice test and an essay 
examination, but that both experimental groups were better than the control group. 

Peterson (1931) used a chemically treated card in conjunction with a felt pen with v*ich the student 
responded. The changing color of the card, when touched by the wet felt, provided knowledge of results 
(blue collect, red incorrect). Adjunct questions were used as guides in reading a selected chapter in a 
psychology text book. The gain scores of experimental group using Jhe chemical cards were, on the average, 
2.4 to 3 times that of the control group. Additional studies conducted in the late 1940*s and early 1950*s 
investigated the use of self-test devices in a variety of applications such as teaching superior students 
(Briggs, 1949; Jensen, 1949), and teaching different subject matter such as Russian and En^sh vocabulary 
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(Stevens. 1953; Scverin. 1955). In most applications, various procedures for providing knowledge of results 
(e.g.. the pundiboard or similar device) witn self-test multiple choice questions were shown to be more 
effective when compared to control groups. 

In a unique application. Meyer (1965) compared an adjunct self instructional program with 
conventional classroom instruction for refresher (review) training in aircraft weapons. Air Crews who were 
required to review this material periodically were used as subjects. Meyer had aircrew instructor personnel 
prepare a comprehensive list of multiple choice questions covering key points in an aircraft weapons 
manual. The questions, including a reference to the page and paragrafA of the manual containing the 
correct answer, were printed in a self-study booklet. One group of subjects (control) received refresher 
training throu^ conventtonal classroom instruction (1 hours. 13 minutes). The experimental group was 
instructed to work throu^ the self-study booklet at least once on their own time during the 11 days 
scheduled for the experiment. Punchboard's vrfiich provided feedback were used by the experimental group, 
and subjects referred to the text only if their initial re^onse as recorded on the punchboard was incorrect. 
Using a 50 item multiple choice criterion test as a pre-treatment test and the same test in reverse numerical 
sequence as a post-treatment test. Meyer found that the experimental group obtained significantly higher ^ 
gain scores than the control group, although the experimental group reported reviewing the materials on the 
average of 4 hours and 13 minutes. To determine whether the higher gain scores of the experimental group 
might be due to the disparity in time spent in studying. Meyer repeated the study holding the time factor 
constant. The control group was given 250 minutes of classroom instruction over 5 consecutive days. Based 
on data from the previous experiment, the control group was allotted 250 minutes for self study . using the 
booklets on their own. Under these conditions, the experimental group stfll scored significantly higher than 
the control group on the post-treatment criterion test. 

In a follow-up study. Meyer (1966) compared four different student response mechanisms. Separate 
groups of Air Force pilots used: (1) chemically impregnated answer sheets; (2) erasable overiays; (3) 
machine presentation devices; and (4) the same punchboard used in the previous two experiments, in 
responding to multiple choice adjunct questions pertaining to a pflot's manual. He found no significant 
differences in gain scores and concluded that they were equally effective. 

Siegd and Fischl (1966) investigated the usefulness of adjunct autoinstruction to teach various civO 
defense concepts to samples of male college students, housewives, and semi-skilled employed women. Two 
modes of presentation were used: (1) conventional reading material, and (2) pre-recorded tape which 
subjects heard throu^ a telephone. Each of these modes had an experimental group (read or heard multiple 
choice adjunct questions after each lesson and receWed feedback from a chemically treated answer ^eet) 
and a control group (receWcd materials twice without adjunct questions). A 72 item muhiple choice test 
was used as the criterion measure. Althou^ the specific results were somewhat different among the three 
samples, those groups using adjunct autoinstructional materials under either mode of presentation 
performed better than the control groups. It was further found that adjunct autoinstruction was most 
effective for less familiar materials. 

Role of Adjunct Questions in Learning. In contrast to the studies described in the previous section, 
recent investigators have become interested in the role of adjunct questions in affecting mediating processes 
in verbal learning. Rothkopf (1966) conducted a study to determine whether adjunct r.uestions have a 
general faclitative effect (enhanced incidental learning) in addition to specific instructional effects, as 
demonstrated by such investigators as Hershberger (1965). and Hershberger and Terry (1%5). Four 
experimental groups read a 5.000 word passage from Rachael Carson's The Sea Around Us, while answering 
constructed response aHjunct questions which differed in location (before and after reading a section) and 
in \y^ether knovdedge of results was given. A fifth experimental group recehred all questions before reading 
the entire passage. The control group did not recehre adjunct questions, but simply read the passage. All 
subjects were allowed only one reading. Two constructed response immediate retention tests were 
administered; one consisting of test items \^ich were identical to those given as adjunct questions (relevant 
learning), and the other consisting of test questions not previously covered by adjunct questio'ns incidental 
learning). Resuhs indicated that the placement of adjunct questions before or after the reading material 
produced significantly higher scores than the control group for the relevant test, but that only the group 
recchring questions after the written material did better on the general test. There were no differences 
related to whether knowledge of results was present or absent. He concluded that adjunct questions have 
general facflitathre effects, when placed after segments ofreading material. The fact that self-test questions 
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improved incidental learning is considered to be evidence that they shape effective inspection behavior 
v^e reading. 

In the foDow on study. Rothkpof and Ksbicos (1967) substantiated the earlier findings in regard to 
the effects of constructed response adjunct question position on incidental learning (immediate retention) 
and provided further evidence of the differential effects of the types of adjunct questions used. Using a 
9,000 word passage from the same book as in the previous study, they found that groups receiving 
questions after reading resulted in hi^er retention on incidental items, but that those receiving questions 
before reading did no better than the control group on either relevant or incidental items. They further 
found that certain types of adjunct questions such as those requiring a name or quantitative response (e.g., 
dates) had more influence on incidental learning than those common or non*technical word responses. The 
authors conclude that adjunct questions, or **test like" events, do not necessarily result in mathemagenic 
behaviors which are instructionally desirable. 

founing (1968) added further evidence on the facilitative effects of adjunct questions covering a 
1,500 word passage describing the characteristics of a fictitious African tribe. After each adjunct question, 
knowledge of results was provided. Using a 2 x 2 factorial design, he compared performance of multiple 
choice test-type (relevant versus incidental) and statement-type (rdevant versus incidental) review item 
groups to a control group on a 26 item multiple choice criterion test administered immediately after 
reading. He found that experimental groups exposed to test*type review items did significantly better than 
either the statement-type review group or the control group. The rdevant review group performed better 
than the incidental review group, but both performed better than the control group. 

Bertou, Gasen, and Lambert (1972) compared the efficacy of adjunct questions (question mode not 
specified) with advanced organizers and post organizers in retention from a televised lecture, as well as, 
reading materials pertaining to atomic ene^ tau^t in a junior high school. Among the three approaches, 
only the adjunct questions facilitated learning as measured by a general test.. 

Watts and Anderson (1971 ) investigated three types of multiple choice adjunct questions differing in 
terms of the extent to whidi responses required application of a psychdogical principle, learned vMc 
reading, to a new situation. They found that the group exposed to application questions scored significantly 
hi^er on the multiple choice immediate retention criterion test for both relevant and incidental items. 

Frase (1967) studied the effects of multiple choice adjunct question location (before or after), 
question pacing (after every 10, 20, or 40 lines), and knowledge of results presence or absence) on the 
retention of information in a 2,000 word passage dealing with the life of William James. Criterion measures 
included question-specific and incidental-general immediate retention multiple choice tests. Results 
indicated that when measured by the question-specific test, questions placed after a moderate length (20 
lines) passage^ with knowledge of results, was most beneficial. For the incidental-general .test, only the 
position of the question (after) was significantly related to immediate retention scores. Knowledge of 
results and question pacing were not significant, although a general trend was observed favoring longer 
passages between adjunct questions.. 

In another experiment using the previous reading materials, Frase (1968) varied question location, 
question pacing and question mode (multiple choice versus constructed response) and measured their effect 
cn immediate retention of specific and incidental materials. He found, as did previous studies, that adjunct 
questions facilitated immediate retention more when they were placed after rather than before the passage 
and that they were more helpful in the retention of specific information than in the retention of incidental 
information. For adjunct questions placed after the passage, retention in general (combined scores including 
both relevant and incidental items) better >^en questions were posed more frequently. Conversely, for 
those placed before the passage, retention was greatest with less frequent questions. For relevant 
information, retention was the same regardless of question pacing, but for incidental information retention 
was best with longer passages (50 lines) between adjunct questions. The implication appears to be that if 
incidental learning is sought (e.g., the adjunct items are merdy representative of the type of information to 
be learned), less frequent adjunctive items may be desirable. The study results indicated no (fifTerences due 
to question mode, althou^ Frase had predicted that the more difficult adjunct format (i.e., constructed 
response) would lead to higher retention. A plausible explanation given by Frase is that because of the 
nature of the question content, the two formats resulted in essentially equivalent difficulty items. 
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The investigations of Rothkopf and his coDeagucs and the formulation of the concept of 
mathemagenic b^aviors (Rothkopf, 1965, 1970, 1972) have generated a great deal of res-^arch in 
replicating their findings using different stimulus materials, as well as in extending their finding? by 
exploring various parameters influencing the effectiveness of adjunct questions. Included in this type of 
research are those pertaining to presentation of materials by different sense modalities; e.g., visual versus 
auditory, (Sanders, 1973; Rothkopf & Hoom, 1970), to various conceptual levels of questions (Allen, 
1970; Watts & Anderson, 1971; ffiller & Denzel, 1973), to repetition of questions (Boyd, 1973), to 
individual differences (Beriiner, Shavelson, Ravitch, & Loeding, 1973; Hollen, 1970), and to the 
effectiveness of ac^unct questions under different incentive levels (Frase, Piitrick, & Schumer, 1970). Still 
others have compared approaches such as prompting (Keller & Cunningham, 1972) and declarative 
statements inserted in the text (Burning, 1968) as compared to adjunct questions. 

In spite of the considerable research interest in Rothkopf s concepts of mathemagenic effects, his 
approach and findings have not been without their critics. Important to the demonstration of the positive 
mathemagenic effects of adjunct questions is their influence on incidental learning. Rothkopfs (1966) 
conclusions regarding the &cilitating effects of adjunct questions in his original study were based on the use 
of multiple t tests between the experimental groups and the control group. Hopkins and Chadboum (1967) 
questioned this approach and reanalyzed Rothkopfs data using the Newman^KeuIs and Dunnett's test and 
found no significant differences, on incidental learning, even allowing for one-tail tests. Ladas (1973) has 
criticized the Rothkopf and Bisbicos (1967) study on the same grounds and provided rationale for the 
inappropriateness of the use of a one*tail test for their experiments. 

Several studies have reported fmdings contradictory to those previously described regarding the 
positive effect of adjurict questions on incidental learning. 

Wadsworth and Flagg (1971) described two studies in which elementary schod subjects, who 
answered multiple choice adjunct questions after reading short segments dealing with Latin history, were 
compared to those who received no questions. Although there were significant differences in one study on 
relevant test items, there were no differences in the second study. In neither study were there any 
differences in incidental learning. 

Swenson and Kulhavy (1973) investigated multiple choke adjunct question position (before or after) 
and question placement (after 1, 5, 10, or 20 paragraphs) on incidental and relevant learning of materials 
describing a fictitious tribe. Althougfi the superiority of ac^unct questions for the rdevant learning was 
demonstrated, there were no significant differences in incidental learning. 

In comparing prompting (underlining key phases) techniques and adjunct questions, Keller and 
Cunningfiam (1972) made the surprising finding that although adjunct question groups performed better on 
relevant test items, they in fact did worse on incidental items than the control group. 

The studies cited above point out certain Actors regarding the role of adjunct questions in learning 
from written materials. First, it appears quite clear that their use enhances learning when measured by total 
test scores and substantiates the eadier findings of Pressey (1950). Second, the position of the questions, 
before or after a reading passage, has differential effects on learning. While pre^uestions appear only to 
effect relevant learning, post-questions, in addition to enhancing relevant learning, are the only ones shown 
to facflitate incidental learning. 

These findings have been interpreted in terms of their role in modifying mathemagenic behaviors 
(Rothkopf, 1965, 1970, 1972) or attentional processes (Anderson, 1967, 1970). Pre^uestions modify 
inspection behavior such that the reader searches for specific information within the written materials. 
Once found, attentional value is lessened. Thus, althou^ specLlc information is retained, there is little or 
no incidental learning. Post-questions provide a general test-taking orientation applicable to reading of 
subsequent paragraphs. It serves not ordy as a ^ecific review question, but also provides an anticipatory cue 
as to the type of material to look for in the following paragraph. 

The effect of other aspects of adjunct questions is not as clear. Researchers have manipulated such 
variables as question pacing, question mode, question type, and knowledge of results to determine the 
effects on immediate retention (Rothkopf, 1966; Frase, 1967, 1968). Rothkopf (1966) concluded that an 
adjunct question presented after approximatdy 1,000 words was optimal while (1967, 1968) found 
more frequent post-questions to be more effective in one study, but less effective in another. Swenson and 
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Kulhavy (1972) demonstrated that questions after Ave paragraphs (330 words) were superior to more or 
less frequent questions on long term retention. Prcssey (1967) encourages questions after meaningful units 
of materials rather than after specific numbers of words or sentences and discusses the advantages of 
end-of-chapter questions. 

The issue of question mode is equally unsettled. Frase (1968) hypothesized that constructed response 
adjunct questions would result in greater retention than multiple choice questions, but fcund no significant 
differences in immediate retention. However, he concluded that the nature of his items did not provide a 
fair test since they were essentially equal in diflTiculty (one word fill-in type). Skinner's (1958) 
requirement for constructed responses in programmed instruction has been tested with conflicting results. 
Evans, Glaser, and Homme (1962) found no differences in performance between overt (active response), 
covert ("thinking" the answer), and reading (answer fflled in) modes. Wittrock (1963) found a significant 
interaction between response mode and mental age among children . Krumboltz and Weisman (1962) found 
no significant differences in inunediate retention, but differences in favor of the overt response group on 
delayed recall. Williams (1963) compared constructed responses, multiple choice responses and a 
combination of the two with grammar school children. Results from a constructed response criterion test 
indicated a relationship between response mode during the course of learning and the use of the same mode 
as a criterion test. Tobias (1973) in his review of the response mode question in programmed instruction 
suggests that when the learning task is technical or difficult, constructed responses are superior to multiple 
choice responses. Holland and Kemp (1965) have offered the "black-out technique" to help explain the 
differences found in comparative studies of overt and covert response modes.. The technique measures the 
extent to which the responses required in a program are dependent upon the content, and determines how 
much of the instructional material can be obliterated (lined-out) without affecting enor rate. In comparing 
12 studies of overt versus covert response modes, they demonstrated that the ones demonstrating 
superiority of overt responses were those which had the lowest "black-out ratio"; i.e., where the fewest 
obliterations could be made without affecting error rate. From the various studies cited above, it would 
appear the effectiveness of response mode is dependent on the learner characteristics, question difficulty, 
and the nature of the criterion measures.. 

Regarding criterion measures, Rothkopf, Frase, Bruning and others typically measured immediate 
retention only ; i.e., experimental passages were read and criterion tests given immediately upon completion. 
More recent investigators have realized the utility of delayed retention measures in research involving 
adjunct questions. Natkin and Stahler (1969) varied subject "arousal" by manipulating the degree of 
pre-exposure to adjunct questions. They demonstrated rather dramatic differences in short and long tenn 
retention between various experimental groups. Swenson and Kulhavy (1973) showed that there was less of 
a loss in retention between immediate and long term criterion test scores for those groups exposed to 
medium length passages between adjunct questions rather than either short or long passages. 

Summaiy 

There have been two broad categories of research in the use of adjunct questions in learning from 
written materials; that of Pressey and his students, and the work stimulated by Rothkopfs concept of 
mathemagenic behaviors. The research conducted by Pressey was characterized by: (1) experimentation in 
the classroom environment over a long period of time (e.g., semester), (2) use of self -test multiple choice 
questions as a test, as well as a teaching aid, and (3) use of an external or ancillary device to provide 
automatic testing, scoring, and immediate knowledge of results. Although little emphasis was placed on the 
nature of the role of the multiple choice questions themselves, Pressey established the usefulness of 
questions as an adjunct to normal classroom materials v/hen used with various devices which provide 
knowledge of results. 

Rothkopf and Frase and others have shown how adjunct questions influence the modificiation of 
mathemagenic behaviors, which in turn contribute to learning. Their research has been characterized by 
experimental manipulations of questions to detennine their effects on verbal learning under carefully 
controlled short term laboratory conditions, using immediate retention tests. 

Although much research has been done, there are still many unanswered issues. As Gagne and Rohwer 
(1969) point out: "More experimental studies are needed. The insertions of questions, or *test4ike events* 
into texts to be studied is a technique that suggests a variety of experimental problems which as yet are 
unexplored: kinds of questions, frequency of questions, amount of questioning." 
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Statement of the Problem 

The improvement of instructiou for education and training is of continual interest to educators and 
psychologists. Much of the current emphasis is on the restructuring of instructional materials and 
presentation techniques to optimize learning. An alternate, and in many ways an equally attractive 
approach, is to explore means of enhancing learning by modifying learner bdiavior, with instructional 
materials as a given. This is particularly true of situations in which text books or reading materials are 
already in existence. 

The overall usefulness of adjunct programming has been demonstrated by Pressey (1950), Meyer 
(1965), Siegel and Fischi (1966). However, tueir approaches involved the use of mechanical or other 
specially constructed devices (e.g., chemically treated paper) to provide immediate knowledge 
of results. Such devices require resources which are normally beyond the routine paper 
reproduction capabilities of a local level training organization. An alternative method is 
simply providing knowledge-of-results by printing the correct answer following a self-test or adjunct 
question. This approach is widely used in conventional programmed instruction, and was selected for this 
study to determine whether the principles of adjunct programming are still effective with this printed form 
of instructional feedback. 

Tne criterion tests, typically used in adjunct programming studies, (Meyer, 1965, 1966; Siegel & 
Fischi, 1966) have measured what would be referred to as relevant (specific) learning. This is consistent 
with the current instructional systems development approach in which specific criterion objectives are 
established and then measured directly to assess learning. Rothkopf (1970) has advanced the notion of the 
positive mathemagenic effects of adjunct questions on incidental learning. Thus far, however, incidental 
learning has been demonstrated only under carefully controlled laboratory conditions. Should this 
phenomenon be shown in a realistic instructional environment using longer reading passages, it would 
represent a serendipitous benefit of adjunct programming which has not been previously advanced by its 
proponents. 

One of the controlled variables in previous studies has been the number of exposures to stimulus 
materials. Almost without exception, subjects were limited to one reading. Predictably, researchers have 
found (e.g., Rothkopf, 1966; Meyers, 1965) that groups without adjunct questions finished reading in 
significantly shorter time than those using adjunct questions. Thus, as Tobias (1973) implies, the argument 
can be presented that the additional time spent studying by experimental groups might account for 
demonstrated differences in learning. The question then arises as to whether adjunct questioiTs are still 
effective if "time-to-study" is held constant. Meyer (1965) has provided some data regarding this issue. 
However, his findings were dependent upon subjects* reports of time spent in study at home. 

The unsettled issues of question pacing and question mode have implications within the context of 
adjunct programming. Previous researchers have manipulated question pacing in terms of the number of 
words or paragraphs between adjunct questions. If adjunct programming techniques are to be applied to 
existing texts, however, it would appear that presenting adjunct questions after each page or chapter (in a 
separate study guide) would require the least amount of alteration to the text. ^ 

In regard to the question mode issue, the data are equivocal. The superiority of constructed response 
questions (in programmed instruction and adjunct questions research) has been shown under certain 
conditions as a function of the time test (immediate versus delayed) and the mode of criterion test 
(multiple choice or constructed response). 

Evidence clarifying the relationship of these two variables in an instructional environment will 
provide further information in regard to the utility of simple adjunct programming techniques to 
supplement ordinary texts. 

The primary purpose of the present study is to explore the effects of three selected variables: (1) 
question pacing (i.e., by the chapter versus by the page), (2) question mode(i.e., constructed response 
ye^us multiple choice), and (3) criterion test mode (i.e., constructed response versus multiple choice) on 
immediate and long term retention. Criterion tests consist of both specific (question relevant) and general 
(incidental) items. 

Specifically, the study is intended to gather further evidence regarding the following questions when 
time of exposure to stimulus material is held constant. 



1 . Do adjunct questions enhance general retention of written materials? 

2. Do adjunct questions enhance relevant learning? 

3. Do adjunct questions enhance incidental learning? 

4. Does adjunct question mode affect retention? 

5. Does adjunct question pacing affect retention? 

6. Does adjunct question mode affect performance as a function of the question mode of the 
criterion test? 

7. Are there differences in the effects of the above variables as a function of whether the criterion 
test is immediate or delayed? 

Answers to these questions could provide further data regarding: (1) the role of adjunct questions in ^ 
learning from reading materials, and (2) the design of adjunct programming materials for use in 
supplementing ordinary text. 



II. METHOD 

Subjects 

The initial pool of subjects consisted of 162 male Air Force Reserve Officers Training Corps (ROTC) 
cadets comprising the entire senior class in a southwestern university. All subjects were enrolled in one of 
nine 4th year mOitary science class sections meeting three hours per week. Four sections met for 1 hour 
three days per week whOe the other five met 1 1/2 hours two days per week. Class sizes ranged from 16 to 
20 students each. Within each class, students were assigned to one of five groups (four experimental groups 
plus a control group). Of the 162 initial pool of subjects, 14 were eliminated from the study because they 
were unable to complete the reading assignment and both criterion tests for a variety of reasons (e g , 
fllness). Three subjects were eliminated on a random basis to equate the number of subjects in each 
treatment group. Thus, the number of subjects used for the final analysis was 145, 29 in each treatment 
group. 

Three subscale scores of the Armed Forces Officer Qualification Test (AFOQT) were collected on 
each subject; (1) Pilot composite; (2) Verbal composite; and (3) Quantitative composite. In addition, 
cumulative grade point averages for all courses taken and a separate cumulative grade point average only for 
ROTC courses were collected. ^ r o j 

Stimulus Materials 

Reading Materials. An Air Force publication entitled The Military Justice System, 1969 revised 
edition, which consisted of 56 pages of text divided into 10 chapters was used as reading material. The total 
™tK"ln''7m ^f/PP^^^*'"^^e>,y 30,000 words. Each page of the manual was in a two<:olumn format 
with 300-^700 words per page, depending on the presence or absence of illustrations and graphs. The 
manuals were unaltered for the experiment with one exception. Each of the manuals used by experimental 
groups exposed to an adjunct question after each page of reading were marked with a red rectangular block 
after each page as a reminder to refer to the adjunct booklet. Experimental groups who were exposed to 
adjunct questions after each chapter and the control group received manuals with the red rectangular block 
after each chapter. The manuals used by the control group were marked as a reminder to record the time 
required to read each chapter. 

The n,1f r'f ^^?'''- "^"^ Obtained from several sources. 

The majority were selected from a pool of multiple choice and constructed response questions which had 
been cons ructed m a previous study. The remainder were constructed by the present invesUgitor to 
provide full coverage of the mOitary justice manual. From the resuhing pool of 150 p i " ons 

9rSli:SV°"''r '^"^ comparaWe constructed response qu'est on s) 

91 were selected for inclusion in the final study. Thirty-two of these 91 questions were reserved for use in 
the criterion test to measure incidental learning and did not appear as adjunct questions The othe 59 nairs 
of questions constituted the adjunct questions used while reading. Thirty-two of these we^pretn ed a'^L 
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m the criterion test to measure relevant learning. The distribution of the 59 adjunct questions ainong the 10 
chapters is given in Table 1. Although a few of the adjunct questions in constructed response format 
required one word responses as in the Frase (1968) study, in the majority of cases, subjects in the present 
study were required to make short phrase or sentence responses. Regardless of the difficulty level of the 
question, however, alternate modes of questions were written to cover the same reading material. The 
following is an example of the constructed response mode of adjunct questions. 

Table 1. Distribution 
of the 59 Adjunct Questions 
(AQs) Among Text Chapters 



Chapter 


AOs 


Chapter 


AOs 


I 


5 


6 


7 


2 


6 


7 


5 


3 


8 


8 


7 


4 


5 


9 


6 


5 


5 


10 


5 



Example L 

Multiple Choice Adjunct Question, 

Article 133 of the Uniform Code of Military Justice prescribes 



a. Conduct unbecoming an officer. 

b. Conditions under which enlisted personnel may be disciplined. 

c. Dress regulations for all military personnel. 

d. What is felony as opposed to misdemeanor. 

Constructed Response Adjunct Question: 

What type of mflitary offense is described in Article 1 33? 

Study Question Booklets. Four experimental adjunct booklets consisting of 59 questions each were 
developed for the present study: 

Booklet 1. Multiple Choice Adjunct Questions by the Page (MCP). This booklet consisted of multiple 
choice questions which were answered after reading one page of text. The booklet was designed such that 
after reading approximately one page in the text, the student turned to the booklet and answered questions 
pertaining to the reading. On the following page of the booklet, the question was repeated and the correct 
answer provided. In addition, reference was given to the page and column of the text where the answer 
could be found. The student was then referred back to the appropriate page in the text to continue reading. 

Booklet 2. Multiple Choice Adjunct Questions by the Chapter (MCC). This booklet contained 
identical questions as in Booklet 1 , except that the students read the entire chapter prior to answering the 
adjunct questions. On the following page, the question was repeated, the answer provided and the reference 
given to the page and column of the text where the answer could be found. Instructions then referred him 
back to the next chapter in the text to continue reading. 

Booklet 3. Constructed Response Adjunct Questions by Page (CRP). This booklet was similar to 
Booklet 1, except that the adjunct questions were in constructed response format. The questions covered 
the same material in the text as Booklet 1 . 

Booklet 4. Constructed Response Adjunct Questions by Chapter (CRC). This booklet was similar to 
Booklet 2, except that the adjunct items were in constructed response format. The questions covered the 
same material in the text as Booklet 2. 

To fllustrate the format of the 4 booklets used in the study, chapter one of each booklet is included 
in Appendices A through D. 
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Criterion Test 



The criterion test consisted of 80 questions divided into 5 subtests of 16 items each. 

Subtest I. Multiple Choice-Relevant (MCR). This subtest consisted of 16 multiple choice questions 
which had previously appeared as adjunct questions in Booklets 1 and 2 (multiple choice adjunct 
questions). 

Subtest 2. Multiple Choice-Incidental (MCI). This subtest consisted if 16 multiple choice questions 
which had not previously appeared in any of the study guide booklets. They covered portions of the text 
other than that covered by adjunct questions. For example, if an adjunct question was drawn from the first 
half of a page in the text, the multiple choice-incidental test question related to materials in the second 
half. 

Subtest 3. Constructed Response-Relevant (CRR). This subtest consisted of 16 constructed response 
test items which had appeared as adjunct questions in Booklets 3 and 4 (constructed response adjunct 
questions). 

Subtest 4. Constructed Response-Incidental (CRI). This subtest consisted of 16 constructed response 
questions not previously used in the adjunct questions. As was the case in Subtest 2 (MCI), the questions 
were drawn from portions in the text not covered by adjunct questions. 

Subtest 5. General Test (GT). These 16 items were selected to test the extent to which information 
learned whfle reading could be applied to different military situations. They were not directed toward a 
specific portion of the text, but required general understanding of basic concepts such as the various levels 
of mflitary courts, forms of restraint, etc., (Items 65-80). 

These subtests collectively covered all chapters in the book and were presented as one 80Mtem test. 
Multiple choice and constructed response test items were alternated. Otherwise, test items were arranged 
randomly. The 80-item criterion test is included in Appendix E. 

Experimental Procedures 

Prior to the first class meeting, an alphabetical class roster was obtained. Within each class, subjects 
were assigned to one of the five treatment groups (4 experimental groups, plus a control group) by 
designating a treatment to an individual, in succession, as their name appeared on the roster.: A packet 
consisting of the instructions, study guide booklet and the appropriately marked text was assembled with 
the subjects name already printed on the instruction sheet. The control group received only the instructions 
and text. 

The experiment was conducted during the normally scheduled segments allocated for mOitary justice. 
Two hundred minutes were allowed for studying the text. Because of the necessity to control the total time 
spent in actual study, the scheme ^own in Table 2 was used to control total reading time. 



Table 2. Reading Time Per Class Period 



Period 




4 Class sections 
(Reading time 
in minutes) 




S Ciass sections 
(Reading time 
In minutes) 


Mon 


Wed 


FrI 


Tues Thurs 


Week 1 


20 


50 


50 


50 75 


Week 2 


40 


40 


Test 1 


75 Test 1 


Week 3 










Week 4 


Test 2 (9 days delay) 




Test 2 (10 days delay) 



At the beginning of the first session in each of the 9 class sections, general instructions were read 
regarding the purpose of the study and the procedures to be followed for the military justice portion of the 
course. A 37-item background information questionnaire was then distributed and completed by all 
subjects. Upon completion of the questionnaire, each subject's packet containing special instructions, text, 
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and study questions booklet (experimental groups only) was distributed. Instructions to those assigned to 
the four experimental groups were identical and included guidance on how to use the booklets. Instructions 
to those in the control group were similar except that there was no reference to booklets. The general 
instructions which were read to all students and the special instructions for the treatment groups are 
mcluded in Appendix F. All subjects were informed that they would be tested upon completion of the 
reading and were aware that test scores would count toward their final grade. All subjects were instructed 
to review materials in preparation for the test, if they completed their reading early. The same experimentar 
read the initial instructions and monitored the classes during the first week in the presence of regular 
instructors. The regular instructors then monitored the classes during the second week and administered 
both criterion tests. 

All reading materials were distributed at the beginning and collected at the end of each class session. 
Since the military justice text was only avaflable through the ROTC department, there is no reason to 
suspect that students had access to any of the materials to study at home. 



III. RESULTS 

The results of the study are presented in two main parts. The first part discusses the overall 
effectiveness of adjunct questions as determined by comparison of the four experimental treatment groups 
with the control group using a 5 X 2 analysis of variance design with three different dependent variables. 
The second part deals with a series of analyses calculated to determine the differential effects of the two 
independent variables with selected criterion variables using a2X2X2X2 within subjects design with the 
last two factors repeated and crossed within levels of the first two factors. Analyses were performed 
separately on short and long term retention. On the latter analyses, the control group scores were excluded. 

Student Characteristics 

The means and standard deviations of the five individual differences measures, by treatment groups, 
are given in Table 3. The first three measures were subscores of the Armed Forces Officer Qualification 
Test: (1) PUot Composite (P), (2) Verbal Composite (V), and (3) Quantitative Composite (Q). The fourth 
was a cumulative grade point average for all courses (CGPA), and the fifth was the cumulative ROTC grade 
point average (CRGPA). A one way analysis of variance was calculated on each of the five measures. With 
the exception of the Quantitative Composite (Q) scores which approached the .10 level of significance, 
none approached the .25 level, indicating that the treatment groups did not significantly differ on these 
variables. 

Table 3. Means and Standard Deviations for Individual 

Difference Measures by Treatment Groups « 











Treatmtnt Groups 








Differences 




MCP 


MCC 


CRP 


CRC 


CG 




Measures 




(n=29) 


(n=29) 


(n=29) 


(n=29) 


(n=29) 


F Ratio 


p 


Mean 


64.66 


62.76 


65.86 


56.03 


67.76 


1.06 




S.D. 


23.75 


22.26 


24.17 


22.89 


24.66 




V 


Mean 


52.59 


52.59 


5534 


44.83 


56.90 


1.14 




S.D. 


22.22 


24.66 


23.56 


233 2 


23.47 




Q 


Mean 


59.48 


50.69 


60.34 


43.28 


5738 


1.93 




S.D. 


28.55 


30.20 


27.64 


24.90 


27.82 




CGPA 


Mean 


2.52 


2.58 


2.48 


2.40 


2.58 


1.00 




S.D. • 


.38 


.36 


38 


34 


.50 




CRGPA 


Mean 


2.88 


3.03 


2.98 


2.97 


2.99 


.43 




S.D. 


.38 


.48 


.46 


.46 


.52 





F. 10 (4,140)= 1.99 
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As a prc-treatnient measure, a 37 item background information questionnaire was administered. 
Several items dealt with demographic characteristics, while others queried study and reading habits and 
outside interests. Preliminary analysis of the data using total test scores to dichot^. ze the group revealed 
no reliable pattern of differences in responses. For this reason, the data were not subjected to further 
analysis. 

Performance Results 

Total Scores. In order to determine the effects of treatment conditions (i.c., four experimental groups 
plus the control group) on ^ort and long term retention, a S X 2 analysis of variance with repeated 
measures on the last factor was calculated. The repeated measures factor consisted of the time of test (i^., 
short and long term). The dependent variable was total scores on each test. 

The means and standard deviations of the 5 treatment groups for both the short and long term 
retention tests are given in Table 4. The summary of the analysis of variance is given in Table 5. Results of 
the analysis indicated that both the treatment group effect (F = 3.18, df = 4/140) and time of test 
(F = 4.57, df = 1/140) were significant at the .05 level. Post hoc comparisons of treatment group means, for 
the immediate and long term retention scores combined, were made using the Newman-Kculs method 
(Winer, 1962). The results, as diown in Table 6, indicated that the experimental groups scores were 
significantly higher (p < .05) than the control group and that the Constructed Response by Page (CRP) 
group scored significantly higher than the Multiple Choice by Page (MCP) group (p < .05). Other pair-wise 
comparisons showed no significant differences. 

Table 4. Means and Standard Deviations for Treatment Groups 
• by Time of Test (Total Scores) 



Tmtm«nt Qroupt 



Time of 
Ttit 




MCP 


MCC 
(n«2S) 


CAP 


cnc 

(n»2f) 


CG 


Short Term 


Mean 


44.44 


47.16 


47.53 


4534 


,41.72 


(Tl) 


S.D. 


6^6 


6.67 


8.73 


7.58 


7.07 


Long Term 


Mean 


45.14 


47.18 


4836 


4635 


43.03 


(T2) 


S.D. 


6.43 


6.78 


6.83 


7.61 


5.46 


MeanTl,T2 




44.79 


47.17 


48.04 


45.85 


4237 



Table 5. Summary of the Analysis of Variance Treatment Groups x 
Time of Test (Total Scores) 



Sourct of Variation 


MS 


df 




Between Subjects 




i44 




A (Treatment Groups) 


28335 




3.18* 


Subject Within Groups 


88.91 


140 




Within Subjects 




145 




B (Time of Test) 


47.85 


1 


437* 


AB 


3.52 


4 


33 


B X Subjects Within Groups 


10.46 


140 





♦p < .05 



The time of test main effect was also significant. Inspection of the treatment group means indicated 
that ail groups (induding the control group) had higher scores on the long term retention test than the 
short term test. The Ueatment X time of test interaction was not significant (F = 33, df = 4/140). 
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Table 6. Ncwman-Kcub Test on Treatment Group Overall 
Means (Total Test Scores) 





CO 


MCP 


CRC 


MCC 


CRP 




V 


1 


IV 


II 


III 


Ordered Means 


42J7 


44.79 


45.85 


47.17 


48.04 


V 




2.42 


3.48 


4.80 


5.67 


I 






1.06 


2.38 


3.25 


IV 








1J2 


2.19 


II 










.87 


=.78r = 




2 


3 


4 


5 


S^q.99(r, 140) 
q .95 (r, 140) 




2.84 


3.21 


3.43 


3.59 




2.16 


2.58 


2.83 


3j01 


V 




• 


** 






I 










• 



IV 

II 



*p < .05 
•*p < .01 

Incidental Learning Scores. In order to detem^'ne the effects of treatment conditions (i.e., four 
experimental groups plus the control group) on short and long term incidental retention, a 5 X 2 analysis of 
variance with repeated measures on the last factor was calculated. The repeated measures factor consisted 
of the time of test. The dependent variable in this case was the scores on the subtest which included the 32 
test items which had not previously appeared as adjunct questions (i.e., MCI and CRI scores combined). 
The scores of the subtest consisting of general items (GT) were excluded. 

The means and standard deviations of the 5 treatment groups for both the Aort and long term 
retention tests are given in Table 7. The summary of the 5 X 2 analysis of variance is given in Table 8. 
Results of the analysis showed that there were no significant main effects for the treatment groups (F = J2, 
df = 4/140), but that there was a significant difference in the time factor at the .01 level (F= 11.24, 
df = 1/140). For all treatment groups, the long term retention test scores were higher than the short term 
scores. The treatment X time of test interaction was not significant (F= J5,df = 4/140). 

Relevant Learning Scores, The effect of the treatment conditions on relevant learning was determined 
by a treatment by time of test 5X2 analysis of variance with repeated measures on the last factor. The 
dependent variable was the scores on the 32 test items which had previously appeared as adjunc'' questions 
(i.e., MCR and CRR scores combined). The general test items were excluded. 

The means and standard deviations of the 5 treatment groups for both the short and long term 
retention tests are given in Table 9. The summary of the 5 X 2 analysis of variance is shown in Table 10. 
Results showed a significant treatment groups effect (F = 8.18, df = 4/140). Pair-wise cciiparisons were 
made using the Newman-Keuls method and are shown in Table 11. Results indicated that scores of all 
experimental groups were significantly higher than the control groups at the .01 level. The Constructed 
Response by Page (CRP) group was significantly higher than the Multiple Choice by Page (MCP) at the .01 
level, and significantly higher than the other two groups (CRC and MCC) at the .05 level. 

Neither the time of test effect (F = .46, df = 1/140) nor the Treatment Groups X Time interaction 
(F = 1 .2 1 , df = 4/140) were significant. 

Experimental Treatment Effects 

In order to determine the differential effects of adjunct question mode (multiple choice versus 
constructed re^onse) and adjunct quesuop pacing (by the page versus by the chapter) on criterion test 
relevance (relevant versus incidental) and criterion test mode (multiple choice versus constructed response), 
a2X2X2X2 analysis of variarKe with within-subjects measures on the last two factors was calculated 
separately for short term and long term retention scores. In these analyses, the control group was excluded. 
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Table 7. Means and Standffd Deviatioiis for Trettmeiit Gro^^ 
by rune of Test (Incidental Learning Scores) 



TIlM of 




MCr 


MCC 


CR^ 


CRC 


cc 


Twt 












*»■») 


Short Term 


Mean 


15.47 


15.44 


15.87 


14.96 


15.49 


(Tl) 


S.D. 


3.01 


3.86 


3.20 


338 


3iO 


Long Term 


Mean 


15.75 


15.93 


16.66 


15.88 


1623 


(T2) 


S.D. 


2.99 


3.54 


2.47 


339 


3.04 


MeanTl,T2 




15.61 


15.68 


1627 


15.42 


15.86 



Table & Su Jmmy of Analysis of Variance Treatment Groups x 
Time of Test (Incidental Learning Scores) 



Souic* of varlition 


MS 


«( 




Between Subjects 




144 




A (Treatment Groups) 


6.04 


4 


32 


Subjects Within Groups 


18i7 


140 




Vnthin Subjects 




145 




B(rmf»eofTest) 


29.89 


1 


11.24** 


AB 


.94 


4 


35 


B X Subjects Within Groups 


2.66 


140 





♦♦p < .01 



Table 9. Means and Standard Deviations for Treatment Groups 
by Hme of Test (Relevant Learning Scores) 



TrMtm«iit Groups 



Tlmo of 




MCr 
(n-2«) 


MCC 
(n«2«) 


CR^ 


CRC 


CO 
(n-2«) 


Short Term 


Mean 


20.61 


22.84 


22.92 


22.18 


18.42 


(TO 


SD. 


3.51 


3.13 


3.98 


3.76 


3.13 


Long Tenn 


Mean 


20.95 


22.00 


2256 


22.86 


18.56 


(T2) 


SJ). 


3.76 


3.12 


3.47 


3^7 


3.43 


MeanTl.T2 




20.79 


22.42 


22.94 


22.02 


18.49 



Table 10. Summary of Analysts of Variance Treatment Groups x 
Time of Test (Relevant Learning Scores) 



Soure* of Varlatlen 


MS 


d» 




Between Subjects 




144 




A (Treatment Groups) 


182.99 


4 


8.18** 


Subjects Within Groups 


2237 


140 




Withm Subjects 




145 




B (Time of Test) 


1.18 


1 


46 


AB 


3.09 


4 


1.21 


B X Subjects Within Groups 


2i5 


140 





••p < .01 
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Table 11. Newman-Xeuk Tests on Tkvttineiit Gioupc 
Means (Relevant Learning Scores) 





CG 




cue 


MCC 






V 


1 


IV 


M 


III 


Order Means 


18.49 


20.79 


?2.02 


22.42 


22.94 


V 




230 


353 


3^3 


4.45 


I 






123 


1.63 


2.15 


IV 








.40 


.92 


n 










JS2 


S. = .39r = 




2 


3 


4 


5 


S. q .99 (r. 140) 




1.41 


1.61 


1.72 


1.79 


S^q.95(r. 140) 




1.08 


1.29 


1.42 


lil 


V 




mm 


mm 


mm 


mm 


I 






m 


m 


mm 



IV 

II 



♦•p < .01 

The first two factors were defined by the crossing of the expenmental treatments (i.e., adjunct 
question mode X adjunct question pacing) and the last factors by the crossing of the criterion test mode 
(multiple choice versus constructed response) and criterion test relevance (relevant versus incidental). The 
latter two factors were tested within levels of the first two; thus, the design wasa2X2X2X2 within 
subjects design with the last two factors repeated and crossed within levels of the first two factors. 

Short Term Retention. The means and standard deviations of the adjunct question mode and pacing 
scores by criterion test mode and test relevance are shown in Table 1 2. The summary of the 2- X 2 X 2 X 2 
with within-subjects mear jres on the last two factors is given in Tal^e 1 3. 

Results showed that there were no significant main effects for either adjunct question mode or 
adjunct question pacing. The question mode X pacing interaction approached significance (5 = 3.21, 
df = 1/1 1 2). but did not reach the .05 level (.05 < p < .10). 

Both within subjects main effects were highly significant (p < .001). Mean scores on multiple choice 
test items (M = 1 1 .76) were considerably higher than on constructed re^onse items (M = 7.02) and mean 
scores on relevant test items (M = 1 1 .97) were higher than on incidental test items (M = 7.72). 

Two 2 way interactions were also significant. The interaction of adjunct question mode (multiple 
choice versus constructed response) X criterion test mode (multiple choice versus constructed response) was 
significant at the .01 level (F = 9.58. df= 1/112). This interaction, shown in Figure 2. indicated that 
although those subjects exposed to constructed response adjunct questions did slightly worse (M = 1 1 j64) 
than tbe multiple choice adjunct questions group (M = 1 1.89) on muhiple choice test items, they did better 
(M = 7.'^5) than the multiple choice adjunct questions group (M = 6.70) on constructed response test items. 
Ihe interaction between test relevance (incidental versus relevant test items) and test mode (multiple choice 
versus constructed response) was also significant at the .01 level. The cell means and standard deviations are 
given in Table 14. This interaction, graphed in Figure 3, indicated that there was a greater discrepancy in 
the number of correct items between the multiple choice and constructed response test mode for incidental 
test items than there was for relevant test items. 

The triple interaction between adjunct question mode X test relevance X test mode was significant at 
the .05 level (F = 5.01 . df = 1/1 1 5). Cell means and standard deviations are given in TaWe 15. This 3 way 
interaction, shown in Figure 4. indicated that those subjects exposed to constmcted response adjunct 
questions scored slightly lower on multiple choice test items for both incidental and relevant tests, than the 
multiple choice adjunct question group, but scored higher on constructed response test items, 
particularly for the relevant test. 

Lortg Term Retention. The means and standard deviations of the adjunct question mode and pacing 
scores by criterion test mode and test relevance arc shown in Table 16. The summary of the 2X2X2X2 
with with in-subject s measures on the last two factors is given in Table 1 7. 
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Table 12. Means and Standard Devialiona of Scores 
of Adjunct Question Mode and Pteing Groups on 
Criterion Test Modes and Rekvanoe Subtests 
(Short Term Retention) 



A4luncl AOiunct 
QiMstion Mode QMtition ftdof 







MuHipIt 
Choiot 


Cowtruclld 
RWPOIIf* 


By 


By 
CtaPttr 


Test Mode 












Multiple 


Mean 


11.89 


11.64 


11.78 


11.74 


Choice 


SJ). 


2.00 


2.04 


157 


2.02 


rorotnicted 


Mean 


6.70 


* 735 


6S4 


7.11 


Response 


SJ). 


im 


2.14 


1.97 


2.15 


TestRdevuoe 












Incidental 


Mean 


7.73 


7.71 


7.84 


7:60 




SJ). 


2.15 


ISO 


1.86 


226 


Relevant 


Mean 


10.86 


11.28 


10.88 


1125 




S.D. 


1.86 


2.16 


2S» 


152 



Table 13, Summaiy of the Andysis of Variance 
(Short Term Retention) 



Source of Variation 


MS 


df 




Between Subjects 




115 




A(AQMode) 


4.54 


1 


.54 


B(AQ Facing) 


.50 • 


1 


j06 


AB 


27.08 


1 


321 


Subjects Within Groups 


8.43 


112 




Wthia Subjects 




348 




C (Test Relevance) 


1300.81 


1 


344.13** 


AC 


5.28 


1 


1.40 


BC 


10.71 


1 


2S3 


ABC 


7.78 


1 


2.06 


C X Subjects Within Groups 


3.78 


112 




D (Test Mode) 


2^3.49 


1 


1,071.40** 


AD 


23.27 


1 


9.58** 


BD 


138 


1 


.57 


ABD 


5.41 


1 


223 


D X Subjects Within Groups 


2.43 


112 




CD 


21.08 


1 


9.58** 


ACD 


11.02 


1 


5X)1* 


BCD 


.82 


1 


37 


ABCD 


.01 


1 




CD X Subjects Within Groups 


2.20 


112 





•p < .05 



**p < .01 



.25 

27 



MCAQ 
CRAQ 




Table 14. Means and Standard 
Deviations for Test Relevance 
X Test Mode (Short Term 
Retention) 



Tttt Mod« 







MC 


CR 


I 


Mean 


10.30 


5.14 




S.D. 


2.24 


1^7 


R 


Mean 


13.22 


8.92 




S.D. 


1.74 


2.25 



15 



(D 

O 
O 
C/) 

§ 

•H 

§ 
♦J 

a> 

PS 



MC Test 
CR Test 



10 



5 __ 

1 



Test Relevance 



Figure 3. Test Mode X Test Relevance Interaction (Short Tem Retention). 
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Table 15. Means and Standard Deviations for Adyunct 
Question Mode x Test Relevance and Test Mode 
(Short Term Retention) 



Subttfts 



Incidental R«l«vanl 







MC 


CR 


MC 


CR 


MCAQ 


Mean 


1038 


5.08 


13.40 


833 




SJ). 


229 


1.95 


1.60 


2.07 


CRAQ 


Mean 


10.22 


5.20 


13.05 


9.50 




S.D. 


220 


1.79 


1.89 


2.42 




4J 



MC CR 
Test Mode 




0 



1 1 


MC 


Test Mode 


CR 



Figure 4. Adjunct Question Mode X Test Relevance X Test Mode Interaction (Short Term Retention). 
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Table 16. Means and Standard Deviations of Scores of Adjunct 
Question Mode and Pacing Groups on Criterion Test Mode 
and Relevance Subtests (Long Term Retention) 



Adjunct A<Uunct 
Question Mod* Qutstion Padng 







MultipM 
Choln 


Con$trueWd 
RMponte 


By 


By 
ChapUr 


Test Mode 












Multiple 


Mean 


11^0 


12.16 


11.99 


11.97 


Qioice 


SJ>. 


2.03 


1.83 


1.86 


1.96 


Constructed 


Mean 


6.86 


7.18 


7.09 


654 


Response 


S.D. 


2.25 


2.13 


2.18 


2.20 


Test Relevance 












Incidental 


Mean 


7.92 


8.14 


8.10 


755 




S.D. 


2.05 


1.85 


1.84 


2.06 


Relevant 


Mean 


10.74 


11.20 


10.98 


10.96 




SD. 


2.19 


2.11 


220 


2.10 



Table 1 7. Summary of the Analysis of Variance 
(Long Term Retention) 



Sourc* of variation 


MS 


df 


F 


Between Subjects 




115 




A(AQMode) 


13.38 


1 


1.63 


B (AQ Pacing) 


.78 


1 


.10 


AB 


17J8 


1 


2.12 


Subjects Within Groups 


8.21 


112 




Within Subjects 




348 




C (Test Relevance) 


1,00536 


1 


330.71 


AC 


1.76 


1 


.58 


BC 


.55 


1 


.18 


ABC 


2.55 


1 


.84 


C X Subjects Within Groups 


3.04 


112 




D (Test Mode) 


2,855.17 


1 


739 .68** 


AD 


.08 


1 




BD 


.55 


1 


.14 


ABD 


.001 


1 




D X Subjects Within Groups 


3.86 


112 




CD 


65.25 


1 


29.79** 


ACD 


12.98 


1 


5.93* 


BCD 


2.07 


1 


.94 


ABCD 


6.90 


1 


3.15 


CD X Subjects Within Groups 


2.19 


112 





*p < .05 
**p < .01 



Results of the analyses of variance revealed no main effects for adjunct question mode (F = 1.63, 
df= 1/112) nor adjunct question pacing (F = .0. df= 1/112). The mode X pacing interaction was not 
significant (F = 2.12, df= 1/112). 



Both within subjects main efTects were highly significant (p < .001). Mean scores on multiple choice 
test items (M = 11.98) were considerably higher than on constructed response items (M = 7.02) and mean 
scores on relevant test items (M = 10.97) were higher than on incidental test items (M = 8.03). 

The 2-way interaction of test relevance by test mode was signiflcant at the .01 level. The cell means 
and standard deviations are given in Table 18. This interaction, shown in Figure 5, indicated the same trend 
observed in the analysis of short term retention. That is, there is a greater discrepancy between multiple 
choice and constructed response test scores for incidental test items than for relevant test items. 

Table 18, Means and Standard 
Deviations for Test Relevance 
X Test Mode (Long Tem 
Retention) 





Test Relevance 



Ftgure 5. Test Mode X Test Relevance Interaction (Long Term Retention). 



The 3-way interaction, between adjunct question mode X test relevance X test mode was significant 
at the .01 level. Cell means and standard deviations are given m Table 19. As shown in Figure 6, the 
interaction effects were consistent with that revealed for the short term retention test. Those subjects 
exposed to constructed response adjunct questions scored slightly lower on* multiple choice test items for 
both incidental and relevant tests, but scored higher on constructed response test items, particularly for the 
relevant test. 
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Table 19. Means and Standard Deviations for Adjunct 
Question Mode x Test Relevance and Test Mode 
(Long Term Retention) 



Subtests 



Incidental Relevint 







MCI 




MCR 


CRR 


MCAQ 


Mean 


10.59 


5.24 


13.00 


8.48 


SD. 


1.96 


2.13 


2.02 


236 


CRAQ 


Mean 


11.17 


5.10 


13.16 


9.25 


SD. 


1.92 


1.78 


1.74 


2.48 



Incidental 




MC CR 
Test Mode 

Figure 6. Adjunct Question Mode X Test Relevance X Test Mode Interaction (Long Term Retention). 
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Aptitude-Tiealment Intenctions (AH) 

Although the investigation of possible aptitude-treatment interactions was not a ^eciflc purpose of 
the study, a post-hoc analysis was performed using the Verbal (V) composite scores and cumulative ROTC 
grade point average (CRGPA). Step-wise multiple linear regression analysis was performed; first with the V, 
CRGPA and mcmberAip in the control group (i.e.. score 1 if in the control group, 0 otherwise) (restricted 
f .lodd), and second, with the addition of the V X CG and CRGPA X CG interaction terms (full modd). The 
dependent variable was total score on the short and long term retention measures. The significance of the 
two interaction terms was tested using the technique described by Ward and Jennings (1973) which involves 
the comparison of the squared multiple correlation coefficients (RSQ's) of the full and restricted models. 
The resulting F statistic was evaluated for significance. The basic hypothesis being tested was that 
differences between total test scores of the experimental and control groups for all patterns of scores on V 
andCRGPA were constant. 

Results indicated that the slopes of the regression of the experimental and contrd groups did not 
differ significantly for the short term retention test. Significant differences in slopes, however, were 
revealed for the long term retention measure. The increase in RSQ at each step in the step-wise regression 
analysis for the full and restricted models is given in Table 20. The differences in the dopes of the 
regression for the experimental and control groups as a function of three levds of verbal composite and 
cumulative ROTC grade point average are shown in Figure 7. The figure indicates that those in the 
experimental group who are high in verbal ability performed better than those in the control group, 
regardless of the CRGPA with total scores increasing as a function of CRGPA. Those in the experimental 
group who are average to low in verbal ability and have low grade point averages, scored lower on the long 
term retention test than those in the control group. 

Tabk 20, Linear Regre^km Analysis of Slope Intenctions 
Between Experimental and Control Groups (Long Term Retention) 



Full Modtl Rtstrlctcd Mod«l 



Sourca 


RSQ 


Sourca 


RSQ 


df 


F 


V 


.1233 


V 


.1233 






CGXV 


.2003 


CG 


.1891 






CRGPA 


.2623 


CRGPA 


.2507 


2/140 


339* 


COX CRGPA 


.2663 










CG 


.2854 











*p < .05 



IV. IHSCUSSION 

The major findings of the study are discussed in two parts. The first part includes a discussion of the 
effectiveness of adjunct questions for short and long term overall retention, as well as for relevant and 
incidental retention. The second part includes a discussion of the differential effects of adjunct question 
mode and pacing on criterion test mode and relevance. 

EffediveiiesB of A<iyunct Questions 

OverdD Effects. When total test scores were used as the dependent variable, all experimental groups 
performed significantly better tfian the control group, as is apparent from the analysis of variance and 
follow-up Newman-Keuls test shown in Tables 5 and 6, respectively. This finding is consistent with those of 
previous researchers who have reported similar results and substantiates the condusions of Ptessey (1950), 
Meyer (1965), Siegel and Fi'xhl (1966), and others regarding the usefulness of adjunct programming 
techniques. The beneficial effects of adjunct programming apparently hold up under conditions of equal 
study time, as well as the substitution of a simple knowledge-of-results approach (as opposed to specially 
constructed mechanisms, e.g., punchboards, chemically treated paper, etc.). 
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Althou^ overall retention was benefited by adjunct questions, further analysis of the relevant and 
incidental learning scores provided additional information regarding the type of learning that was enhanced. 
Analysis of variance using incidental learning scores, ^own in Table 8, revealed no significant differences in 
incidental learning between experimental groups and the control groups, while the differences in relevant 
learning, Aown in Tables 10 and 11 , were hi^y significant. These results indicated that althou^ adjunct 
questions enhanced learning as measured by total retention scores, most, if not all, of the difference was 
attributable to relevant learning. Thus, the results of the present study support previous findings regarding 
relevant learning, but do not support previous findings (Bruning, 1968; Frasc, 1967, 1968) regarding the 
positive effects of adjunct questions on incidental learning. 

fiftuisig that much, if not all, of the beneficial effects of adjunct questions were specific to 
relevant >.ai ' Ing is not inconsistent with the views of Skinner (1958) and Presscy (1967) and the current 
trend in instructional design v^ch stipulates the development of specific instructional objectives measured 
by direct test items. The concept advances the basic notion that if a point is important enough to be tested, 
it should be highlighted during learning. 

Although the lack of demonstrable differences in incidental learning may appear to be in 
contradiction to Rothkopfs concept of mathemagenic behaviors, the differences in experimental 
conditions and incentives* must be considered. Rothkopf (1965), Frase (1967, 1968), Bruning (1968), and 
others investigated the role of adjunct questions under very carefully controlled laboratory conditnns and 
allowed subjects only one reading of the stimulus materials. In adutwn, subjects were normally volunteers 
reading neutral materials which had little incentive value since the subject's performance had no bearing on 
his grade. In the present investigation, conditions were quite different. All subjects were given the same 
amount of time to study materials covering a required academic topic. Althou^ it was not specifically 
aimounced, subjects assumed that the test they would take would contribute to their final grade. Thus, 
motivation to learn was probably quite hi^. Frase, Patrick and Schumer (1970) have presented evidence 
that mathemagenic behaviors may be relatively ineffective under high motivation^ levels. 

The significant differences in short and long term retention on total scores and incidental learning 
scores was to be expected. However, the fact that long term retention scores inaeased for all treatment 
groups was somewhat surprising and contrary to most previous findings, which have demonstrated 
significandy lower scores on long term retention measures (Sanders, 1973; Keller & Cuimin^iam, 1972; 
Berliner etal., 1973). 

An obvious explanation of the increase in scores is the effects of rehearsal and the opportunity to 
review test questions with other students after the short term test. The fact that the identical test was used 
twice could have provided an opportunity for practice which resulted in higher long term retention scores. 
This is somc^i^iiat substantiated by the fact that significantly higher long term retention effects were not 
revealed for relevant learning, but only for incidental learning scores. Thus, repeating items for the third 
time (first as an adjunct question, second as a short term retention test, and third as a long term retention 
test) did not increase learning. However, the fact that incidental items were initially exposed during the first 
test and repeated for the second time during the long term retention test, resulted in increased learning for 
an groups including the control group. It would appear that most of the benefits of rdiearsal may be 
realized after one exposure. 

In spite of the plausibility of the rehearsal or practice explanation, the increased scores on long term 
retention are still puling. Hrst, the subjects in the experiment were not aware that they would be taking 
the criterion test twice. Further, the stimulus materials (study guide booklets and manual) used in the 
experiment weie not available outside of the classroom. Tlius, the subjects had neither the motivation nor 
means to review after the first test. Secondly, the length of time between the first and second test (9-10 
days) was longer or as long as those in other studies (e.g., Keller and Curmingham, 5 days; Sanders, 7 days, 
using identical tests). In Sanders' study (1973), only one4ialf of the subjects took the immediate retention 
test and all subjects took the long term retention test. He did find rehearsal effects in that those taking the 
test twice had higher scores on the long term retention test than those taking only the long term retention 
test, but their scores were, nevertheless, lower on the delayed measure. The hi^ motivation of the 
subjects in the present study and the fact that study time was equal for all groups, in addition to the 
rehearsal opportunity, may be partial explanations for the discrepancies between the present and previous 
studies. This area requires further investigation. 
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Effects of Adjunct Question Mode and Pacing. To detennine the eflfects of adjunct question mode 
and pacing, analyses were nnade separatdy on short and long term retention scores, excluding the control 
group. The analyses are summarized in Tables 13 and 17. The lack of main effects for the independent 
variaWes indicates that under the conditions of the present study, the mode of adjunct question used O-c., 
multiple choice versus constructed response format) and the pacing O-e., presenting adjunct questions by 
the pag? or by the diapter), had little differential effects on retention. TWs result, when considered in light 
of the flnding that all experimental groups performed significantly better than the control group, indicates 
that the presence of adjunct questions after reading enhances learning and is not greatly influenced by the 
mode or pacing of the questions. The implication is that, for materials of the type used in this study, 
multiple choice questions printed at the end of a chapter with provisions for knowledge of results are as 
efTective for general retention as either mode of adjunct question presented at shorter intervals. 

However, the interaction of adjunct question mode and criterion test mode (found in the analysis of 
short term retention. Table 13), indicates that constructed response adjunct questions may be superior, 
depending on the performance required in the criterion test. If the test performance desired is that of recall 
(as represented by constructed response tests) rather than recognition (as in multiple choice tests), the use 
of constructed re^nse adjunct questions appears more efTective. Althou^ this interaction did not appear 
in the analysis of the long tcfrm measures, the three way interaction in both the ^ort and long term 
analyses indicated the superiority of constructed response adjunct question groups in answering constructed 
response test items, particularly for relevant materials. The multiple choice adjunct question group, in all 
cases, performed sli^tly better on multiple choice test items. These findings substantiate those of Williams 
(1963) who found very similar results when investigating response modes in programmed instruction. 

Aptitude-Tieatment Intenctions 

The results of the step-wise multiple linear regression analysis are given in Table 20 and illustrated in 
Figure 7. The plot of the significant interaction indicated that regardless of the verbal ability scores, those 
students with higher cumulative ROTC grade point averages (CRGPA's) profited most from the use of 
adjunct questions. However, for ti^ose with medium to low verbal abilities who also have low CRGPA's, the 
use of adjunct questions appeared to be somewhat detrimental. Hie impUcatioi^ is that adjunct questions 
are most beneficial for those individuals whose past achievement has been rdatively hi^. The use of 
adjunct questions by those ^o have had a record of low achievement is questionable. 

These results add some further data to those reported by Hollen (1970) and Berliner et al. (1963). 
HoDen found that the performance of those high on associative memory was impaired by the use of adjunct 
questions. Beriiner et aJ., reported data indicating that subjects with high vocabulary scores ternl to perform 
better without adjunct questions. To the extent that CRGPA's are considered an aptitude measure, the 
findings of the current study are somewhat contrary to those cited above. However, an CRGPA is probably 
more an index of motivation and past achievement. Further exploration of the relationships between 
aptitude measures, past achievement and mothration as related to the use of adjunct questions would be of 
interest. The cunent findings are too prdiminary to have generalizability . 



V. SUMMARY AND CONCLUSIONS 

The purpose of this study was to: (1) investigate the effectiveness of adjunct questions, presented 
within the context of adjunct programming, in enhancing general retention of written materials, and (2) 
explore the differential effects of adjunct question mode (multiple choice versus constructed response) and 
adjunct question pacing (by the page versus by the chapter) on short and long term retention when reading 
time is held constant. 

A review of the literature indicated that there have been two main categories of relevant research: (1) 
that dealing with the overall effectiveness of adjunct programming, consisting basically of adjunct 
questions with provisions for immediate knowledge of residts« using a variety of subject matter, and (2) 
that stimulated by the work of Rothkopf dealing with how adjunct questions influence the modification 
of matfiemagenic behaviors, which in tum contribute to learning. 

The review showed that previous demonstration of the usefulness of adjunct programming involved 
the use of mechanical or other specially constructed devices to provide immediate knowledge-of-results. 
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The approach taken in the present study was to provide knowledge-of-results by simply printing the answer 
and text page reference on the page inunediately following the adjunct question. 

The previous research on mathemagenic behaviors has been^ypified by the manipulation of adjunct 
question position, mode, pacing, etc., under carcfuDy controlled laboratory conditions using short term 
retention meaisuTCs. Experimental subjects were allowed only one exposure to stimulus materials resulting 
in different lengths of time spent by experimental and control groups in studying the materials. The 
approach taken in the present study was to control **time-to-study*' in a realistic classroom environment 
using short and long term retention measures. 

As a result of the review of literature, the following questions were devdoped for siudy under the 
conditions specified above: (1) Do adjunct questions enhance general retention of written materials? (2) Do 
adjunct questions enhance relevant learning? (3) Do adjunct questions enhance incidental learning? (4) 
Does adjunct question mode or pacing affect retention? (5) Df>es sdjunct question mode affect 
performance as a function of the question mode of the criterwn lest? (6) Are there differences in the 
effects of the above variables as a function of whether the test is immediate or delayed? 

To investigate these questions, 4 adjunct question booklets were developed with knowledge of results 
provided on the page following the adjunct question. The 59 questions in each booklet covered the main 
points in an existing text used in a regularly taugjit ROTC subject. Four experimental groups (with 29 
subjects each) used the booklets while reading the basic text. A control group (n - 29) read the text 
without the booklet. All subjects were allowed 200 minutes during normally scheduled classes to study the 
materials. The 80-item aiterion test consisted of five subtests. Four of the subtests varied in question mode 
(multiple choice versus constructed response) and question relevance (incidental versus relevant). Incidental 
questions were those which had not previously appeared as adjunct questions. Relevant questions were 
those whkh had been used as adjunct questions. The fifth subtest consbted of general items designed to 
measure the ablity of subjects to apply information learned from reading to practical situatfons. The total 
test was administered twice; one day after the last reading session and again 9 to 10 days later. 

A 5 X 2 repeated measures design anda2X2X2X2 within subjects design were used as the basic 
approaches to the analysis of the data. 

The major conclusions of the study may be summarized as follows: 

1. Adjunct programming is an effective technique as a supplement to existing texts. The beneficial 
effects apparently hold up under conditions of equal study time for control groups and the substitution of 
? simple knowledge-of-results approach to provide instructional feedback. 

2. The beneflcial effects of adjunct questions are limited, in large part, to the learning of relevant 
materials or those materials to which a ^eciflc adjunct question has been addressed during learning. 

3. There is no evidence that adjunct questions enhance incidental learning, under the conditions of 
this study. Unless the beneficial effects of adjunct questions for the retention of incidental materials can be 
demonstrated in the classroom environment using actual teaching materials, the concept of mathemagenic 
behaviors is not meaningful for practical applicatk)n. 

4. There is an interaction effect between adjunct question mode and test question mode. If the test 
performance desired is that of recall rather than recognition of relevant materials, constructed response 
adjunct questions iippear to be superior to muhiple choice questions. 

5. For the type of reading materials used in the present study, placement of adjunct questions at the 
end of the chapter is as effective as interspersing questions after each page. 

6. The effects of adjunct questions remain relatively stable when measured by short and long term 
retention tests. 
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NAME 



SSN 

CLASS § SECTION 
COLLEGE MAJOR 



MILITARY JUSTICE SYSTEM 
STUDY QUESTIONS 

— KmET — 



er|c , 
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CHAPTER I 
QUESTIONS AND ANSWERS 



READ ALL OF CHAPTER I BEFORE 
TIMING MS PACE. 



45 



CHAPTER I 
(Pages 1-4) 

1. The power to make rules for the government and regulation of 
naval forces is vested in 



a. the President. 

b. the Congress. 

c. the Joint Chiefs of Staff. 

d. the Supreme Court. 

2. The Fifth Amendment to the Constitution as interpreted by the 
Supreme Court holds that the only type of offenses that are subject 
to military law are those that are . 

a. premeditated. 

b. capital. 

c. committed on foreign soil. 

d . servi ce * connected . 

3. The U.S. Articles of War were first revised in 1874, vtfien 

was (were) specifically prohibited. 



a. trying. of officers by civil courts. 

b. cruel and unusual punishment. 

c. striking of officers by enlisted men. 

d. all of the above. 

4. Tne 1949 Manual superseded the Manuals of the 1920' s, and provided 
for 



a. bad conduct discharges. 

b. Judicial Councils. 

c. defense counsels with equal or better qualifications than 
trial counsels. 

d. all of the above. 

5. Which is the best reason for a separate code of military law? 

a. Some behaviors are considered offenses in civilian life, but 
not in the military. 

b. The seriousness of a military offense depends upon the 
immediate situation, vtfiich is not covered in civil codes. 

c. Military duties impose certain rules of conduct in addition 
to rules governing civilian conduct. 

d. Certain persons in the Armed Forces are not subject to U.S. 
Civil Law. 



CHECK ANSWERS 
ON NE?Cr PAGE 
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Chapter I 



ANSWERS 

1. The power to make rules for the government and regulation of 
naval forces is vested in the Congress (b) . 

Page 1, Column 1. 

2. The Fifth Amendment to the Constitution as interpreted by the 
Si4)reme Court holds that the only type of offenses that are sub- 
ject to military law. are those that are service -connected (d) , 

Page I, Column 2. 

3. The U.S. Articles of War were first revised in 1874, when cruel 
and unusual punishment (b) were specifically prohibited. 

Page 2, Column 1. 

4. The 1949 Manual superseded the Manuals of the 1920 's, and pro- 
vided for bad conduct discharges, Judicial CoiJicils_|^^and defense 
counsels with equal or better qualifications than trial counsels" 



Page 3, Column 1. 

5. Which is the best reason for a separate code of military law? 

Military duties impose certain rules of conduct in addition 
to rules governing civilian conduct (c) . 

Page 4, Column 1. 

RECORD THE TBE: 



NOW TURN TO CHAPTER 2 AND 
CONTINUE READING 
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Appendix B 

Sanple Study Guide Booklet. Multiple Choice Adjunct 
Questions By Page 
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NANE 

SSN 

CLASS § SECTION 
COLLEGE mJOR 



MILITARY JUSTICE SYSTEM 
STUDY QUESTIONS 



DY QUEST] 
"BOOKLET 



48' 49 



CHAPTER I 



START READING CHAPTER I BEFORE 
TURNING mis PAGE 
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SO 



CHAPTER I 
(Page 1) 



1. The power to make rules for the government and regulation of 



naval forces is vested in 

a. the President. 

b. the Congress. 

c. the Joint Oiiefs of Staff. 

d. the Supreme Court. 



2. TTie Fifth Amendment to the Constitution as interpreted by the 
Si4)reme Court holds that the only type of offenses that are 
siiject to military law are those that are . 

a. premeditated. 

b. capital. 

c. conmitted on foreign soil. 

d . senace - connected . 



CHECK ANSWERS 
ON NEJfT PAd. 
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Chapter I 



ANSWERS 



TTie power to make rules for the govermient and regulation of 
naval forces is vested in the Congress (b). 

Page 1, Column 1. 



Tlie Fifth AmendTnent to the Constitution as interpreted by the 
Si9)reme Court holds that the only type of offenses that are sub- 
ject to military law are those that are service-connected , (d) 

Page 1, Column 2. 



NOW TURN TO PAGE 2 OF THE 
MANUAL AND CONTINUE READING 
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QJAPTER I 
(Pages 2-4) 



The U.S. Articles of War were first revised in 1874, vrfien 
was (were) specifically prohibited. 

a. trying of officers by civil courts. 

b. cruel and unusual punishnient. 

c. striking of officers by enlisted inen. 

d. all of the ^ove. 



The 1949 Manual sqperseded the Manuals of the 1920*s, and 
provided for . 

a. bad conduct discharges. 

b. Judicial Councils. 

c. defense counsels with equal or better qualifications 
than trial counsels. 

d. all of the above. 



CHECK x\NSWERS 
ON NEXT PAGE. 
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Chapter I 



ANSIVERS 



The U.S. Articles of War were first revised in 1874, when cruel 
and unusual punishment (b) was specifically prohibited. 

Page 2, Column 1. 

The 1949 Manual superseded the Manuals of the 1920' s, and 
provided for bad conduct discharges, Judicial Councils , 
and defense counsels with equal or better qualifications than 
trial counsels (cj ^ 

Page 3, Coluim 1. 



GO Mac A>ro REi^ 

BEFORE TURNING 
THIS PAGE. 
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aiAPTER I 
(Page 4) 



Which is the best reason for a separate code of military law? 



a. Some behaviors are considered offenses in civilian life, 
but not in the military. 

b. The seriousness of a military offense depends upon 

the immediate situation, which is not covered in civil codes. 

c. Military duties inpose certain rules of conduct in addition 
to those rules governing civilian conduct. 

d. Certain persons in the Armed Forces are not subject to U.S. 
Civil Law. 



CHECK A NStVER 
ON m^f PAGE. 
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Chapter I 

ANSWER 

Which is the best reason for a separate code of military law? 

Military duties impose certain rules of conduct in addition to 
those niles governing civilian conduct . (.cJ 

Page 4, Column 1 
RECORD THE TIME: 



NOW START READING CHAPTER 2 IN THE MANUAL 
BEFORE TURNING THIS PACT 
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Appendix C 

Sample Study Guide Booklet. Constructed 
Response Adjunct Questions By Chapter 
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NAME 

SSN 

CLASS § SECTION 
COLLEGE MAJOR 



MILITARY JUSTICE SYSTEM 

STUDY QUESTIONS 
BOOKLET 
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CHAPTER I 
QUESTIONS AND ANSWERS 



READ ALL OF CHAPTER I BEFORE 
TURNING THIS PAGE. 



fl 
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CHAPTER I 
(Pages 1-4) 



What power, especially important for military law, is granted to 
Congress imder Section 8, Article I of the Constitution? 



How does the Fifth Amendment as interpreted by the Supreme Court 
affect the Military Justice System? 



Our First Articles of War (1806) were taken from the British 
Articles of War. IVhat major change was made to tlie Articles in 
the 1874 revision? 



The 1949 Mmuai superseded the Manuals of the 1920's, and addi- 
tional clianges were made in the Military Justice System. 
Describe at least three of the changes. 



The Armed Forces clearly need a separate Military Justice 
System. One reason is that the Amied Forces operate throughout 
the world. What is another? 



CHECK ANSWERS 
ON NEXT PAGE 
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Chapter I 
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ANSWERS 

1. What power, especially important for military law, is granted to 
Congress under Section 8, Article I of the Constitution? 

Answer Congress is empowered to make rules for the government 
and regulation of land and naval forces. 

Page 1, Column 1. 

2. How does the Fifth Amendment as interpreted by the Supreme Court 
affect the Military Justice System? 

Answer The Fifth Amendment insures that only service -connec- 
ted offenses in the aimed forces will be subject to 
military law. 

Page 1, Column 2. 

3. Our First Articles of War (1806) were taken fran the British 
Articles of War. What major change was made to the Articles in 
the 1874 revision? 

Answer Cruel and unusual punishments were prohibited. 

Page 2, Columnt 1. 

4. The 1949 Manual superseded the Manuals of the 1920*' s, and addi- 
tional changes were made in the Military Justice System. Describe 
at least three of the changes. 

Answer Legislation provided for: (1) enlisted persons as court 
members in appropriate cases, (2) placed a lawyer on the 
court as a law member, (3) trial and defense counsels 
had to be similarly qualified, (4) petitions for new 
trials were established, (5) criticism of courts by com- 
manders was prohibited, (6) bad conduct discharges were 
introduced as a type of punishment, (7) review bodies, 
higher than the Board of Review were established 
(Judicial Councils) . 

Page 3, Column 1 

5. The Armed Forces clearly need a separate Military Justice System. 
One reason is that the Armed Forces operate throughout the 
world. What is another? 
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Answer Another reason is that some actions are crimes in the 
Armed Forces but are not ciimes in civilian life. 

Page 4, Column 1. 



RECORD "mE TIME: 



NOW START READING CHAPTER 2 



\ 



ERIC 



62 



Appendix D 

Sample Study Guide Booklet. Constructed 
Response Adjunct Questions by Page 



ERIC 



63 ^3 



NAME_ 

SSN 

CLASS § SECTION 
(DLLEGE MAJOR 



MILITARY JUSTICE SYSTEM 

STUDY QU ESTIONS 
IT 



ERIC 



64 

64 



CHAPTER I 
QUESTIONS AND ANSWERS 



START READING CHAPTER I 
BEFORE TURNING THIS PAGE 
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CHAPTER I 
(Page 1) 



What power, especially iii^wrtant for military law, is granted to 
Congress under Section 8, Article I of the Constitution. 



How does the Fifth Amendment as interpreted by the Supreane Court 
affect the Military Justice System? 



OffiCK ANSWERS 
ON NEXT PAtai 




Chapter 1 



ANSWERS 



What power, especially in5>ortant for military law, is granted to 
Congress under Section 8, Article I of the Constitution. 

Answer Congress is en5)awered to make rules for the goveiTiinent 
and regulation of land and naval forces. 

Page 1, Column 1. 

$ 

How does the Fifth Amendment as interpreted by the Sipreme Court 
affect the Military Justice System? 

Answer The Fifth Amendment, as interpreted by the Supreme Court, 
insures that only service -connected offenses in the 
armed forces will be subject to military law* 

Page 1, Column 2, 



NOW TURN TO PAGE 2 OF TOE 

MANUAL m ccmmt readtmg 
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OiAPTER 1 
(Pages 2§3) 



3. Our First Articles of War (1806) were taken frcjm the British 
Articles of War. What major change was made to the Articles in 
the 1874 revision? 



4. The 1949 Manual superseded the Manuals of the 1920' s, and addi- 
tional changes were made in the Military Justice System. Describe 
at least three of the changes. 



CHECK ANSWERS 
OM NECT PAGE 
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Oiapter 1 



ANSWERS 



Our First Articles of War (1806) were taken frcan the British 
Articles of War. What major change was made to the Articles in 
the 1874 revision? 

Answer Cruel and unusual punishments were prohibited. 

Page 2, Colunn 1 



The 1949 Manual superseded the Manuals of the 1920* s, and addi- 
tional changes were jnade in the Military Justice System. Describe 
at least three of the changes. 

Answer Legislation provided for: (1) enlisted persons as court 
members in appropriate cases, (2) placed a lawyer on the 
court as a law member, (3) Trial and defense counsels 
had to be similarly qualified, (4) petitions for new 
trials were established. , (5) criticism of courts by 
coraoianders was prohibited, (6) bad conduct discharges 
were introduced as a type of punishment, (7) review 
bodies, higher than the Board of Review were established 
(Judicial Councils). 

Page 3, Column 1 



GO BACK AND READ IN THOE 
MWm (PAGE 4) BEFORE 
TuRNTNG THIS PAGE. 
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aiAPTER 1 
(Page 4) 



The Airaed Forces clearly need a separate Military Justice System. 
One reason is that the Aimed Forces operate throu^out the world. 
What is another? 



(HECK ANSWER 
ON NEXT PAGE 
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Chapter 1 



ANSWER 



The Armed Fbrces clearly need a separate Military Justice System. 
One reason is that the Aimed Forces operate throughout the world. 
What IS another? 



Answer Another reason is that some actions are crimes in the 
Armed Forces but are not crimes in civilian life. 

Page 4, Column 1 



RECORD "mE TIME: 



NOW START READING CHAPTCR 2 

BER3RE rmmn mis p>age~ 
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